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ABSTRACT 
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compute :r-b a sed system significantly improved learning ability, an 
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research. (RL) 
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A computer-based aid to self-directed learning has been developed 
to meet this need. This aids system is implemented on the PLATO system 
and uses the touch-panel capability of the PLATO-IV terminal. Students 
are presented with a task which requires complex learning, and they are 
given considerable information much more than is needed, in fact 
to attain the task. The aids system is designed to allow students to 
break down their task into a set of more easily attained objectives, to 
decide when information is relevant to their objectives, and in general 
to monitor their progress toward achieving the task. 

The complete training aid is quite complex, so that students are 
trained in its use over a numbar of sessions. New features of the system 
are introduced in alternate sessions, and students then practice with the 
system using a new learning task. This task ia each case requires the 
student to troubleshoot or debug a simulated device. This device produces 
output, some of which is defective, and the student is required to locate 
the faulty component by examining the defective output and by reading an 
on-line "technical manual" for the device. 

A pilot experiment has been completed to allow a formative evaluation 
of the self-directed aids system. Although the results of this experiment 
found no statistically significant differences between the treatment groups, 
they suggested directions for future research. 
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SUMMARY 

On-the-job training requires considerable independence on the part 
of the trainee. Unlike a student in a classroom, the trainee must arrange 
information resources in such a way that he can learn how to perform his 
specific task without wasting valuable time reading irrelevant information. 
He must further direct this learning himself. 

A computer-based aid to self-directed learning has been developed 
to meet this need. This aids system is implemented on the PLATO system 
and uses the touch-panel capability of the PLATO-IV terminal i Students 
are presented with a task which requires complex learning, and they are 
given considerable information — much more than is needed, in fact — 
to attain the task. The aids system is designed to allow students to 
break down their task into a set of more easily attained objectives, to 
decide when information is relevant to their objectives, and in general 
to monitor their progress toward achievinrj the task. 

The complete training aid is quite complex, so that students are 
trained in its use over a number of sessions. New features of the system 
are introduced in alternate sessions, and students then practice with the 
system using a new learning task. This task in each case requires the 
student to troubleshoot or debug a simulated device. This device produces 
output, some of which is defective, and the student is required to locate 
the faulty component by.examining the defective output and by reading an 
on-line "technical manual" for the device. 

A pilot experiment has been completed to allow a formative evaluation 
of the self-directed aids system. Although the results of this experiment 
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found no statistically significant differences between the treatment groups, 
they suggested directions for future research. 
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A F0W4ATIVE EVALUATION OF A COMPUTER-BASED INSTRUCTIONAL 
SYSTEM FOR TEACHING JOB-ORIENTED READING STRATEGIES 



I. INTRODUCTION 

Learning from textbooks differs significantly from learning in real- 
mrjd situations such as on-the-job training. Information in a textbook 
is. arranged in such a way that a student is led in an idealized fashion to 
build on earlier knowledge. That is, the textbook writer first presents 
elementary information, then more complex information (based on the ele- 
mentary-information), then still more complex information, and so on. In 
the real world, however, information is not so neatly arranged. Complex 
concepts are frequently encountered before the more elementary concepts 
upon which they are based. Worse yet, there is generally far too much infor- 
mation available, much of it totally irrelevant to what the student wishes 
to learn. Since no one has pre-arranged and pre-digested the information 
for him, the student trying to learn in a real -world sttuation must take on 
those responsibilities himself. He must, that is, be self-directed in his 
learning. 

Not infrequently, inexperienced technicians find themselves assigned 
to jobs in which they have to maintain equipment or systems they have not 
seen before or may have encountered only briefly in school. They have. In 
such conditions, a strong need to learn more about these devices, using 
available technical documents as a source of information. The technical 
manuals they consult may, in some instances, presuppose prior knowledge 
that is incomplete or partially forgotten. Thus, the technician on the 
job may have a requirement to learn information at several lower levels 
of complexity as well as at the technical manual level and to organize a 
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sequence of acquisition. The information he heeds may not be contained in 
a single place iji any document, and the structure of the document— table 
of contents, index, and so on— may not help him locate the proper infor- 
mation. Under these circumstances, technicians who are not self-directed 
might try to read the entire technical document from cover to cover, obviously 
wasting valuable time. On the other hand, some technicians might ignore the 
information resources available and simply begin sticking test probes into 
the defective equipment, equally obviously wasting time. In either case, 
the technician would benefit from knowing some techniques of being self- 
directed, of determining which information is relevant to his specific task 
and learning only that information. 

This paper describes initial steps toward the development of a 
training system to help people faced with this kind of complex learning 
task. Our research plan calls for several cycles of development and 
testing of the training system. In this report we discuss our first 
pass at the development of such a system, the training system is de- 
scribed and the results of a pilot experiment on the effectiveness of 
the system are reported. This formative evaluation will be used to 
revise the self-directed learning system described below. 'The revised 
system will be tested agaiff on college students and then revised for 
use in technical: "training context's. New data bases appropriate to such 
contexts will be created, and the system will be tested in this context. 
A summative evaluation will be performed. 
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Computer-based Aid to Self-directed Learning 

Our training program, is designed to teach students, how to use: a 
computer-based aid to self-directed learning that has been developed 
in our laboratory. A learning task is presented to a student^ and he 
is given considerable information— too much information, in fact— to 
complete his task. The aids system is designed to aTlow the student 
to break down his task into a set of more easiT . a 3 bbjactives, 

to^decide^en^-^chapter^of-^he-technical manual-^ is--^ ^ 

objectTves , and^i n general f 0 keep^tl^ck^ of *"Ri s l¥arni n g . Tfn s ai c!s ^ ^ ^ ^ 
system can be thought of as consisting of a nuirt)er of "pages," each of 
which presents certain types of information and provides the user with 
certain options. The four major components of this system are the Task 
page, the Objectives page, the Concents page, and the Relevant Contents, 
page. (The term "page" in this context indicates one or more screen dis- 
plays on a PLATO- IV panel). From any of these pages, the student can 
choose to go to any one of the others. The major components and their 
subcomponents are shown in Figure 1. 

The Task page states the overall task or learning goal for the studient. 
The task changes for each session that the student uses the aids system, but 
in each case it involves learning enough material to troubleshoot a defective 
device of some kind. (See Section III. below). The task page also gives the 
student access to the example output from the defective device. The 
student uses this output to help determine the source of the fault in the 



We thank Steve Cheney for advice in the initial stages of the design 
of the aids system and for help in recruiting students to test early ver- 
sions of the system. 
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TASK 
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DEVICE 
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4. SPECIFY DEPENDENT 
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ED INFO ^SOURCES 



X 



INFORMATICH^ SOURCES 
(TECHNICAL MANUAL) 
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FIGURE 



1. Automated Aid to Self-Directed Learning, Functions of Components SpeciTiedT 
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device. When the student feels he is ready to attempt the task, he can go 
to a test accessible from the Task page. This test requires that the student 
identify the faulty component of the device; if he fails this test, he is 
sent back to the aids system to study additional material/ He can later 
return to the Task page to attempt the task once again. Another important 
funrti on accessible fif^om the Task page is jthe student's goaT stack, an overt 
representation of dependencies which the student discovers among the various 
'^aS?E>"bT'1Efie^lM 
sources. 

The goal stack the student sees on the PLATO terminal screen looks 
something like the diagram in Figure 2. The arrows in the goal stack dia- 
gram show dependency relationships that hold among the student's objectives 
and the Information resources available. For example, the curved line from 
objective 1 to objective 2 means that objective 2 cannot be attained until 
objective 1 is first attained; objective 2 is thus dependent on objective 1. 
Similarly, the line from information source 7 to objective 4 means that 
objective 4 requires the understanding of information source 7 for its 
attainment. The curved line from information source 3 to information source 
1 means that 1 Is dependent on 3; 3 should therefore be studied before 1. 

The student is taught several heuristics to help him use the goal 
structure effectively. For example, if the node on the goal tree that rep- 
resents a particular goal has an arrow head impinging on it, then that goal 
should not be attempted until the goal at the other end of the arrow has been 
attained. This is a simple restatement of the principle that it is better 
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to attempt the prerequisites of an action before attempting the action. 
Since the goal structure keeps a record of goals attained by means of 
check marks' next to completed goals, this rule is easy to heed. 

The second major component of the self-directed learning aid is 
the Objectives page. The primary function of this page is to maintain a 
List of .the. learning .objectives based on the, task at hand. From the 
Objectives page the student can formulate new objectives that he or she 
belleyes^ace^neces'sar^^to^the^acicompXl^^ 
-has entered an-objeetiveHt wi-Ll -be- Mste 

he returns to that page. Two other functions available on the Objectives 
page are checking off objectives that have been attained (by reading the 
relevant information) and specifying dependency relationships between object 
ives. When the student utilizes the latter option, the Aids system records 
the fact that there is a dependency between the two> objectives named by the 
student. ^ This dependency is shown whenever the student chooses to look 
at his Goal Stack (accessed from the Task page), and an arrow is drawn 
from the required to the dependent objective on the Goal Stack. Thus, the 
arrows between any two objectives on the Goal Stack page are determined by 
what the student has done on the Objectives page. 

The Contents page simply provides a lisTTf^Tne'Tfnes of^'c]^^ " 
or information sources of the technical manual that covers the device tliat 
the student is troubleshooting. The student can scan this list of titles 
and make decisions about the probable relevance to his objectives of some 
Of the topics mentioned. When he decides that the material under a certain 
title is likely to be relevant to some objective, he exercises the Choose- 
Title option from the Contents page. Picking a title has the effect of 



throwing control immediately to the Matching page. On the Matching page, 
the student is shown the list of objectives and the title he "just picked; 
he must specify which of those objectives requires that he learn the 
material named by that title. If he does not want to match any of his 
objectives with the chosen title, then he must cancel his choice of the 
title^. iif ht wishes,, he can then go to the Objectives page,..make up a 
new objective, and then return to the Contents page to select the title 
agairb^pjanningj;^^ 



^the^student. j s^.encouraged-to.^sel ect ^only^ those^titl es^:he ^needs^to^-sol-ve- ^ 
his problem. As a result of the choices made on the Matching gage, the 
Aids system remembers which of the chosen information sources are required 
by which of the objectives. This information appears whenever the student 
decides to look at his Goal Stack. It determines the arrows that- are drawn 
from the numbers of the relevant information sources to their objectives. 
(See Figure 2). . 



TASK 



OBUKTIVK^ 



INFORMATION 
SOURCES 




FIGURE 2. A Sample Goal Stack 
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Choices made on the Contenjts pages have one other consequence". Those 
titles that are chosen as relevant to some objective (and are matched with 
the objective) will appear on the Relevant Contents paqe. The Relevant 
Contents page; is the student's personalized table of contents relevant to 
the troubTeshooting problem he is trying to solve. Whenever the student 
chooses to go "to the Relevant Contents page, he sees a l ist of all these - 
chosen titles. A number of functions are available from this jage., .. Hirst r 
^Re~n*uaefrrT:aTrdee'idir'to"re'ad"^ listed there-. 

SecondV if a student has reatf and urulerstdod an infofmfiM ToQf^^^ 
check off the title on the Relevant Contents page to signify that this sub- 
goal was attained. If the student has read an information source and dis- 
covered that it was irrelevant, he can decide to remove it from the list of 
relevant information sources. A fourth option available is to specify 
dependencies between information sources. For example,, if the student de- 
cides that relevant information source i cannot be understood until relevant 
information source 1 has beenjunderstood, then he can specify that 1 is 
. dependent upon 3^. 

The last three choices listed above all have consequences for the 
Goal Stack. If the student has checked off a title, then that title's 
number will have a check mark below it in the Goal Stack. If a title has 
been removed from the list because it is irrelevant, then it will not appear 
in the Goal Stack at all. And i.f a dependency between- two information 
sources has been specified, then an arrow will connect the numbers of 
their titles in the Goal Stack. 

The information sources or "chapters" themselves are quite simple. 
Each consists of a number of pages through which the student can progress. 

-8- 
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The student can page through the information sburce either forward or 



backward. In addition, from ai^ page the student can elect to return 
to the Relevant Contents page. ^ 
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^ If _ • * V 

_ ; . n. JTIAINING SEQUENCE 

Students do not inmediately begin with the complete self-directed 
learning aids system as it was discussed in the previous section. Instead, 
they are led to that version In a series of training sessions, each one 
having . more of the. features discussed in Section I tlian thje, prey 1 pus 
session. The complete training sequence is shown in Figure 3. 



Session 
1. 

2. 
3. 

4. 
5. 

7. 



FAMILIARIZAf ION WITH TERMINAL 
INTRODUCTION TO SIMPLIFIED AIDS SYSTEM 



PRACTICE WITH SIMPLIFIED AIDS SYSTEM 



INTRODUCTION TO MORE COMPLEX AIDS SYSTEM 
(WITH GOAL STACK). 



PRACTICE WITH MORE COMPLEX AIDS SYSTEM 



INTRODUCTION TO FINAL AIDS SYSTEM 



PRACTICE WITH FINAL AIDS SYSTEM 



POST-TEST WITH FINAL AIDS SYSTEM 

(NEW TASK mmiH) . 



FIGURE 3. Training Sequence for Use of Self- Directed Aids System 
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The initial session familiarizes the student with the PLATO terminal 
and introduces him to the- aids system. The.student begins by playing a 
few games, of. tlc-tac-toe against the computer to introduce him to the 
idea of touching the terminal panel. He then goes through a training 
lesson ;that teaches him to use the most rudimentary version of the aids 
system. ,(Thi s version dif fens. from, the f ul.l system, discussed ,i n„Section.-I, 
in the following ways: first, the student cannot formulate his owh.objecr . 
tives, but must make use of a. set of objectives.-prpv.ided; second,, the. sys-=- 

tem does not prov/ide a Goal- Stack; thirdy- the- Objects ves -page-dpes -riot-™ 

provide the option of making explicit the dependencies among the objectives; 
fourth, the Relevant Contents page does not provide the option of making 
explicit dependencies among chosen titles.) This first session has three 
parts. In the first part, the student is taught about the overall, structure 
of the Aids system and is given a quiz on his understanding of the system. 
Those students who score below the criterion must repeat this section of 
the lesson. In the second part of the lesson, the student is taught about 
the specific functions of each of the components of the system.. This section 
of the training requires that the student step through each of these func- 
tions in a simulation of their actual use. In the third part of this 
session, the student has his first opportunity to practice with the limited 
aids system on a very simple task (learning to use the PLATO keyboard to 
type and edit answers). 

In the second session students are required to solve a troubleshooting- 
problem through the use of the simplified Aids system they learned: about 
in the first session. The task is quite complex, and most students require 
from one to two hours to accomplish it. 



The third session introduc»^s students to a more complex Aids system. 
To the simplified system they have already learned about, the Goal Stack 
is added. In addition, the options to specify dependencies among objectives 
(on the bb.iectives page) and to specify dependencies among information 
■sources (on the Relevant Contents page) are included. The lesson requires 
-the-stUdent -to make appropriate responses in a-simulation of the functions 
of these new options. 

In the- fourth session, the students practice* with' this more complex ^ 
Aids system; They are requi red* to troubleshoot the"":same*^type"'of device' 
that they have already had a troubleshooting probl on on, but the problem 
and its symptoms are new. 

The fifth session introduces the student to the writing of his own 
objectives on the automated Aids system. When this lesson has been com- 
pliBted, the student has been introduced to the complete Aids system depicted 
in Figure 1. This lesson is quite short and is usuaTTy combined with that 
of the sixth session for one long session. 

In the sixth session, the student practices with the complete Aids 
system. The new troubleshooting task is, again, on the same type of device 
as were all the previous tasks. 

The seventh session is a post-test session, although from the student's 
point of view it is simply another practice session with the full Aids sys- 

tern. In this session, the troubleshooting task is on a defective devite of 

J 

a different type from that with which the student is familiar. New irw'orma- 
tlori resources are, of course, provided. I 



III. TASK DOMAINS FOR THE TRAINING SYSTEM 



Each time a istudent practices with some version of the Aids system, he 
must solve a complex learning task* In each case, the task is to trouble- 
shoot or debug a defective device, this device produces output, s.ome of 
J^hich il loAoiaect by/ examining, thl^^ .and. by ^reading.infor^ation.^ ^ 

sources on the various components of the device, a student can determine 
which cpmpo^^^^ Is faulty (that is, causing, the improper output K^^^E^^ 
,pra,c:tim .session., wi.th. the Aids .system.has^a..di.fferent .task--a-di fferent--^^-^ 
component is faulty, and, therefore, different symptoms, are -presented, 
each time. 

Two such devices were selected as task areas, a sentence generator 
and an essay generator. (These devices are simulations, not physically 
embodied machines.) The sentence generator was used in sessions 3 through 
6, and the essay generator was used in session 7, as a post-test of the 
training. These task domains were chosen to conform to a number of cri- 
teria: 

(1) The topic matter permits the construction of "debugging" 
or troubleshooting problems. This is important because the topic matter 
i¥^l> BFlmalog^^^^ ^ 
front Naval electronics technicians. 

(2) The topic matter is sufficiently difficult that it could 
not be easily and completely comprehended by a single reading of a simple 
"technical manual" (the information sources). Again, this feature is an 
important part of the analogy to learning about the imaintenance of elec- 
tronic equipment. 
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(3) The topic matter is sufficiently simple that no special 
technical, scientific, .or mathematical skills or knowledge are prerequi- 
site to an understanding of the "technical manual." This feature is an 
important concern for the recruitment of subjects. Ideally, a large class 

of subjects should be available f^ whom the task tqpic j's equally MfanijJASr. 

(4) The topic matter is one with which the investigators are 
sufficiently familiar that they can easily prepare suitable technical 
documents. ' _ _^ , i „ 

The Sentence Generator 

The major components of the sentence-generation device are shown in 
Figure 4. A. given component is comprised of a series of sub-components. 
Arrows in the diagram show the flow of control in the device. Where there 
are choice-points in the production of a sentence, this is represented 
by the use of switches in the diagram. For example, within the Noun-Phrase 
Generator, there is a three-way choice among three sub-components of 
.Noun-Phrase. These are called NPl, NP2, and NP3. Only one of these serves 
as the activation of the Noun-Phrase Genfrator at one time. Within the 
conponent galled NP3 there are more switches signifying other options in 
the production of a noun phrase with this Noun-Phras"e Generators -^If the 
NP3 unit is activated, then the DET unit must function; the Modifier-Phrase 
Generator either may or may not be called upon. A dashed- line box surround- 
ing a component (such as the NP3 within NPl, or the NP within the 
Prepositional -Phrase Generator) signifies that control is surrendered to that 
component (defined elsewhere in the diagram) at that point. When that 



embedded component has finished running, control returns to the exit 
point of the dashed-line box. The presence of the embedded Noun-Phrase 
Generator c(Miiponents in the Noun-Phrase Generator, and that of the embedded 
Sentence Gene^^^^ the VerbrPhrase Generator makes the 

definitely recursive. The two optional components after the VerbrPhrase 
Generator permit the application of the Dative-Shift and Passive trans- 
formations to the output of the rest of the Sentence Generator. 

The diagram in Figure 4 is a functional analogue to a set^of produc-' 
tion rules which generate hierarchical structures, plus two transformational 
rules. The rules equivalent to the component diagram are given below (see 
next page). These rules describe a powerful device which produces a wide 
variety (although not all) of the grammatical sentence types in English. 
It provides a rich area for troubleshbotihg or debuyging problems. 

A student assigned to troubleshoot the Sentence Generator has access 
to a technical document containing 25 chapters, each several pages long; 
10 of these chapters discuss various aspects of sentence grammar but are 
not relevant to the Sentence Generator itself. The remaining 15 chapters 
describe the functions and interrelationships of the components of the 
Sentence Generator. A list of all titles of these information resources 
is given below (Page 18). 



Phrase- structure Rules 



1. 


S 




NP' + VP 


2. 


NP 




NP, , NPg, NP3} 


3. 


NP^ 




NP3+ Relpro + VP 


4. 


NPg 


— 


PN 


5. 


NP3 




Det + (MP) + N + (PP) 


6. 


MP 


— * 


(Adv) + Adj 


7. 


PP 


— * 


P + NP 


8. 


VP 




VP,. VPg, VP3, VP4} 


9. 


VP, 




Vi 


10. 


VP2 




+ NP 


11. 






y'^ + NP + Datprep + NP 


12. 


VP4 




V^ + Comp + S 



Transformational Rules 



1. Passive SD: NP - V^ - NP 



1 2 3Zj> 

SG: 3, was_ + 2 + en , b^;. + 1 



2. Dative Shift SD: V^j + NP + to + NP 

1 2 3 4Zj> 
SD: 1, 4, 2, 0 

Symbols 

S - sentence, NP = noun phrase, VP = verb phrase, Relpro = relative pronoun, 
PN = Proper noun, Det = determiner, MP = modifier phrase, N = noun, PP = 
prepositional phrase, Adv = adverb, Adj = adjective, Vi = intransitive verb, 
Vt = transitive verb, Vd = double transitive verb, Datprep = dativl prepositi 
Vc « complanentizing verb, Comp = complementizer, en = perfective marker. 
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INFORMATION RESOURCES AVAILABLE FOR 
SENTENCE GENERATOR TROUBLESHOOTING 

1. The Dative-Shift Component 

2. Dependency Grammar 

3. The EQUI-NP Transformation 

4. Finite-State Grammar 

■5. ti near and Hierarchical Structure - 

6. The Mod-Phrase Generator 

7. The Noun-Phrase Generator 

8. The NPl Component 

9. The NP2 Component 
TO. The NP3 Component 

Tl. the Particle Movement Transformation 

12. The Passive Component 

13. Phrase-Structure Grammar 

14. The Prep-Phrase Generator 

15. Rearrangement Transformations 

16. The Sentence Generator 

17. Syntactic Trees 

18. The Transformational Component 

19. Transformational Grammar 

20. The Verb Deletion Transformation 

21 . The Verb-Phrase Generator 

22. The VPl Component 

23. 7he VP2 Component 

24. The VP3 Component 

25. The VP4 Comporient 
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The technical document contains a list of grammatical: sentences 
generated by a fully-functioning device. {In addition, the discussion 
of each component contains representative examples of phrases or words 
correctly generated by that component.) As examples of sentences gener- 
ated by the device in Figure 4, consider the list below: 

The quarterback passed the ball to the tight endi 

i 

Professor Hotchkiss is sleepiYig. 
A very naive freshman bought the wrong booki 
The teaching assistant who gave the coed ah A+ 
was visited by the dean* ^ 
Harry thinks that Frank graduated. 

A student in the J>ack row cgughed,.. _ \ 
The instructor realized that the students were shoring. 
Other chapters of the technical document, which discuss other com- 
ponents of the sentence generator, also present lists of sample possible 
outputs for those components. For example, the "Noun-Phrase Generator" 
chapter lists a sampling of grammatical noun phrases, such as 
the tight end 
a very naive freshman 
Harry 

the teaching assistant who gave the coed an A+ 
When subjects are pres^anted with their tisk, they see a similar list 
of sentences, but some of these sentences are ungrairmatical due to the 
failure or malfunction of a particular component of the sentence-generating 
device. For example, consider the following list. (An asterisk (*) before 
a sentence indicates that it is ungrammatical .) 

o -1&- 
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*The textbook was written by. 
The dean sent a letter to the department heads. 
A Student who failed the exam is crying. | 
That extremely young freshman surprised the professor. 

*The secretary in the chairman's office discovered that 
taught yoga. 

The trophy was presented to the team by the chancellor. 
(English speakers are sometimes clever enough to provide .a setnantic interpre- 
tation of a sentence marked as ungrammatical. The point is that the sentence 
is nonetheless ungrammatical according to-the-grammar/device given to the 
student-) In this case, the defective component is NP2 (alternatively, 
phrase-structure rule 4),. which failed to. output proper ^nouns.. ;One way of 
showing the nature of the defect is ^presented in the diagram in Figure 5, 
As can be seen in this drawing, the NP2 component is "OTpty.'* It has no 
effect, other than to permit exit from the Noun-Phrase Generator without 
producing a noun-phrase, whenever the second position of the highest-level 
switch in the Noun-Phrase Generator is chosen. When the student correctly 
selects NP2 as the faulty component, he has solved his task. 

The Essay Generator 

The second device, an Essay Generator, is depicted in Figure 6. The 
Essay Generator is supposed to produce well-formed essays on a variety of 
topics. It accomplishes this end, in theory, by the sequential; aetivation 
of a number of its components. A given component ordinarny contains a^ 
number of subcomponents. Arrows in the diagram show the flow of control 
in the device. Where there are choice points in the production of ah 

32-20- 



NOUN-PHRASE GENERATOR 



NPl 









Relpro 











NP2 



NP3 



»" ^DET[ -»^ 



Mod-Phrase Gen, 




ADV 



ADJ] 



Prep-Phrase Gen. 



FIGURE 5. Diagram of a Faulty Noun-Phrase Generator 
Component of the Sentence Generator Device. 
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essay, this is represented By the use of switches in the diagram.. For 
example, within the Topic Organizer, there is a three-way choice anK)ng three 
subcbmppnents of the Topic Organizer. These are called the Chronological 
'Organizer,, the Causal Organizer, and the Exemplification Organizer. There 
is a four-way switch within the Paragraph Generator, the setting of which 
determines whether a given paragraph will be produced by the CTaim-and- 
Evidence Unit, the Event-and-Reactibns Unit, the Compare-Contrast Unit, or 
the Princi pi e-and- Inference Unit. Students assigned to accpmplish trouble- 
shooting tasks based on the Essay Generator have accsss to a technical: 
manual of 109 pages on the PLATO system. This technical manual consists 
of twenty-one chapters, fourteen of which discuss the functions apd:int^^ 
relationships of the components of the Essay Genera tor and contain exam^^^^^ 
of the outputs of the various components and of the entire system when it 
is functioning properly. Which of these chapters are relevant depends, 
of course, upon the specific troubleshooting task encountered by the student. 
(The other seven chapters contain general information about writing but have 
nothing to do with troubleshooting the device; thus, these chapters are 
always irrelevant.) The list of all titles of the information sources is 
given on the next page. 

An example of the output of the Essay Generator when it is functioning 
properly is given below. } 

Some of the Effects of Watergate 

The Watergate scandal generally refers less to the actual 
break-in at Democratic headquarters than to the later attempts 
to cover up White House involvement in the planning of the 
operation. The discovery of this involvement and subsequent 
widespread publicity had a number of far-reaching effects. 
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INFORMATION RESOURCES AVAILABLE FOR 
ESSAY GENERATOR TROUBLESHOOTING 



t. 


Causal Organizer 


2. 


Chronological Organizer 


3. 


Claim-and-Evidence Unit 


4. 


Compare-Contrast Unit 


5. 


Counter 


6. 


Elaboration Unit 


7. 


Essay Generator 


8. . 


Event-and-Reactions Unit 


9. 


Exemplification Organizer 


10. 


Figures, Graphs, and Illustrations 


n. 


Footnotes 


12. 


Headings 


13. 


Paragraph Generator 


14. 


Parts of Speech 


15. 


Principle-and-Inference Unit 


16. 


Reference Citations in Essays 


17. 


Sentence Characteristics 


18. 


Sentence Producer 


19. 


Sentence Types 


20. 


Topic Organizer 


21. 


Topic Sentence Unit 



One result was the eventual resigrtatibn of the President 
of the United States while under threat of impeachment. This, 
in turn, meant i hew administration with a Targely new cabinet. 

Another effect of Watergate is that the public has 
become very suspicious of its elected officials. In the 
last election, being an" incumbent or having political experi- 
ence often' seemed" to. be^^^^ liability than an advantage 
to a candidate." Gitizens- are "^suddenly quick to demand 
e>(plahatiohs for any improprieties. 

A third effect has been a change in the relative- s^^^^^ 
of the Republican and Democratic parti bs.fh^ Republicans have 
lost i^bership, while the Democrats have gained. The Republican 
party Treasury, which had had a surplus,, is now. In the red. The 
Democratic treasury, by contrast, had been deeply in the red but 
has since almost fully recovered. 

When subjects are presented with the task of debugging the Essay Generator, 

they see several such essays, but some of them are defective due to ^the failure 

-or malfunction of a particular component in the Essay Generator; The nature^'" 

of the defect depends upon the type of component that is defective. For 

example, if some component within the Topic Organizer is faulty, then the 

paragraphs within an essay might appear in a random order rather than the 

orders specified by those components. If the defect lies in a component of 

the Paragraph Generator, then the sentences within a paragraph might appear 

in the wrong order. For example, consider the following essay. 



Questionable "Scientific" Theories 

Recently a number of questionable theories have been 
proposed by scientists working outside their areas of specialty. 
In many cases these theories have been avidly adopted by large 
segments of the public. Yet scientists in the fields that 
dear with these theories are often skeptical of the claims made. 

The public, however, has respbnded"favorably to Professor 
Bandersnatch's numerous appearances on television talk shows 



-25- 

18 



and- has purchased over V.5 million copies of his book. Pyramid 
People . Archaeologists and Egyptologists iii particular have 
greeted his claim vfith hoots of derision. One example of this 
phenomenon was the reaction to Professor Arnold Bandersnatch ' s 
announcement that the. ancient Egyptian pyramids are actually 
the remnants of ancient spaceships to Earth. 

However, since the appearance of Talmay.'s book. Hair Oil , 
In September of last year, sales of Vitamin E in .this country 
have Increased }50%. Doctors and biologists. have, almost uni- 
versally scoffed at this idea. Another example is the claim 
made by the .physicist Elmirfalmay that Vitamin £, taken in 
large doses, will prevent hair loss. 



In this case, the defective component Is the Event-and-Reactions Unit. 
All paragraphs of this type have scrambled sentence order, with reactioiis 
to some event appearing before the statement of that event. All other 
paragraph types are correct, however. 
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IV. COGNITIVE MODEL FOR SELF-DIRECTED LEARNING 



One way of viewing the goats of this research is to say that we intend 
to :find: the means to teach people how to do effective web-learning (described 
In Norman, 1973, 1974, in^press). What is it that they will know when they 
have. graduated from our training procedures? How will what they know guide 
their learning of complex materials in the future? 

Our answers to these questions are couched in terms of schema-theory 
(Norman, Runmlhart, & LNR, 1975; Rumelhart & Ortony^ in press; Munro' & 
Rigney, 1977). The central tenet of schema- theory Is that knowledge guides 
thought. Stated bjaldly, this seems to be a truism. In schema-theory, how- 
ever, explicit claims, are made about the means by which knowledge guides 
thought. Computer simulations of schema-theory models provide rigorous 
tests of the adequacy \of the proposed mechanisms for the relation of con- 
cepts in memory (of knowledge). Knowledge, in turn, to a large extent, 
consists of "frozen" or fossilized activations--copies of other concepts in 
memory,-with specific details determined by the particular contexts, within 
which those concepts were activated (see Munro & Rigney, 1977, for further 
'explanation)^ — — — - - — 

In schema-theory terms, the knowledge that subjects acquire as a result 
of the training described elsewhere In this report is best represented in 
terms of a prescriptive schema . A prescriptive schema is a conceptual struc- 
ture, which, when activated, gives people the impression that they are giving 
themselves instructions. Prescriptive schemata are responsible for the effects 
that we attribute to "self-direction strategies." The set of schemata that 
students acquire from our training program is an abstract conceptual struc- 
^ure-with considerable scope. (The uses of -the terms ^-abstractness** and 
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"scope" with respect to schemata are discussed in Munro & Rigney, 1977). Here 
are. the hypothesized schemata that we believe students acquire as a result 
of their training. 

(1) SELF-DIRECTED-LEARNING (TASK) 

is when - 
BUILD-GOAL-STRUCTURE (TASK) 
TASK-PURSUE (TASK) 

end. 

(2) BUILD-GOAL-STRUCfURE (TASK) 

is when 

ANALYZE (TASK, for OBJECTIVES (TASK))^ 
PREREQUISITE-SEARCH (for EACH (OBJECTIiVE) , in OBJECTIVES) 
PREREQUISITE-SEARCH (for EACH (OBJECTIVE),, in CONTENTS) 
end. 

(3) TASK-PURSUE (TASK) 

is when - ■ 

EXAMINE (GOAL-STRUCTURE) 

UNTIL (CHECKED (EVERY (OBJECTIVE)), PURS^^^^^ _ 
TASK-AHEMPT (TASK) 
end.^ 

(4) TASK-AHEMPT (TASK). 

is when 

IF (DO (TASK), then QUIT, else SELFr'OIRECTED-LEARNING (TASK)) 
end. 
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The ANALYZE sub-schema has not yet been represented. How people are 
able to discover the prerequisites or component actions^ of a task is not 
well understood. 
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(5) . PREREQUISITE-SEARCH (for GOALS, in SUBGOAL-SET) 

is when 

FOR-EAGH (MEMBER, of SUBGOAL-SET, 

IF (PREREQUISITE (MEMBER, for GOAL), 

then (SPECIFY-DEPENDENCY (MEMBER, to OBJECTIVES-LIST)))) 

end. 

(6) PURSUE (GOAL)^ 

is when . 

FOR-EACH (SUBGOAL (NECESSARY (SUBGOAL, to GOAL))* in GOAL-STRUCTURE. 
WHILE (ANY (UNSATISFIED (SUBGOAL' (NECESSARY (SUBGOAL', to 
SUBGOAL)))), 

PURSUE (SUBGOAL')) 
TRIAL (SUBGOAL)) 
end. 

(7) UNSATISFIED (GOAL) 

is when 
NOT (CHECKED (GOAL)) . 

NOT (ELIMINATED (GOAL)) ' ^ 
endV 

(8) TRIAL (GOAL) 

is when 

ATTEMPT (GOAL) to ATTEMPT (ACTION, of GOAL) 
EVALUATE (GOAL) 
end. 



This structure is a variant of Rumelhant~& Ortony's (in press) schema 
for TRYing, a subschema of their PROBLEMrSOLVING schema. 
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(9) EVALUATE (GOAL) 
is when 

IF (NECESS^-HY (GOAL, to HIGHER-GOAL), 

then IF (SATISFIED (GOAL), then CHECK (GOAL). 

else TASK-PURSUE (TASK)), 
else ELIMINATE (GOAL, from GOAL-STRUCTURE)) 
end. 

01 0) AHEMPT (GOAL) ~ 
is when 

IF (BELIEVE (CAUSE (ACTION, SATISFIED (GOAL))), 
then DO (ACTION), , 

else when SUCCEED (PREREQUISITE-SEARCH (for GOAL) ) , , 
AHEMPT (PREREQUISITE (GOAL))) 

end. 

According to the first of these schemata, the student believes that the 
way to achieve a task through self-directed learning is first to build a goal 
structure and second to pursue the task, using that goal structure. The 

second schema listed above describes what. Is, inyoly^^ goal 

structure. One analyzes a task for objectives (subgoals necessary for the 
performance of the task), then one searches for prerequisite relationships 
among these objectives, between the available information resources and 
the objectives, anc; among the relevant available information resources. 
However, the schema does not contain explicit reference to the process of 
adding these relationships to the goal sJbructure,.Mcaus.e J;he goal sAructure 
is constructed for the student by the program that aids him or her in self- 
directed learning. The fifth schema listed above is an essential part of 
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the goal -structure-building schema, since it specifies how the search for 
prerequisites is conducted* 

The second major part of self-directed learning, after building a goal 
structure, according to the above schemata, is to pursue the task. The 
third schema above gives the. top-level structure for task pursuit. One 
examines the newly constructed goal structure first; then one pursues the 
objectives included in that goal structure until every one of them has been 
checked. (Checking is^ the process by which a student marks the attainment 
of a subgoal, using the aids program oh PLATO). When all the necessary 
objectives have been checked, the student attempts the task. If the attempt 
fails (see schema #4), then he begins the self-directed learning process 
again, reconstructing or modifying the goal structure. 

The pursuit of objectives is governed by the sixth schema, given above. 
This is a recursive procedure that traces down ^dependency relationships in 
the goal structure. When a goal is found that has ho prerequisitesi that 
goal is subjected to a trial. This means (see #8, 9, & 10) that the student 
does an action to bring about the goal and then evaluates the. results of 
that action. If the goal is satisfied, he checks the goal and then pops 
back to the appropriate point in the procedure that is pursuing an objec- 
tive. If it is not satisfied, he looks for a new way to pursue his over- 
all task. If the attempt reveals that the goal was unnecessary to the 
attainment of its higher goal, then it is dropped from the goal structure. 

The above schemata constitute working hypotheses about the nature of 
the conceptual changes brought about by training in the self-directed 
learning aids program discussed above. 
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The prose explanations of these schemata, above, emphasize the way in 
which these schemata cull each other in a tpprdown, conceptually-driven 
processing mode • Naturally, there is also a bottpm-up, data-driven aspect 
to the activation of these schemata in normal circumstancies:. For example, 
when a student finds that he has satisfied a goal Csay, as a result of 
reading one of the relevant information resources), this activates the subr 
schemata in the fourth Tine of the ninth schema presented above* The 
activation of these subschemata (IF (SATISFIED (60AL):i then CHECK-( GOAL i....) 
activates, in a data-driven fashion, its "parent^ schema i .EVALUATE* the 
activation of EVALUATE, in turn, can activate the schema that calls it, and 
so on, so that activation spreads in an upward as well as a downward direc- 
tion. 
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I V. EXPERIMENT' 

An experiment was conducted to test the effects of the self-directed 
learning iaids system. A control condition was established, containing only 
the Task and Contents pages of the system described in Section 11. A student 
in the control condition has the same learning task and the same Infoma t ion 
to read, but he has none of the Aids system available to a student in the 
experimental condition. (Information sources in the control condition, 
are accessed directly from the Table of Contents, As soph as the student 
touches a title, he is shown the corresponding information source); 

Control Training Sequence 

t 

The training sequence for control subjects is similar to that for experi 
mental subjects, except that the basic system is never modified for them, so 
that there is no need for teaching sessions other than the Initial cfae. 
Consequently, all sessions arei practice sessions using, the control system. 
The complete sequence is shown in Figure 7. 



1 ' 


1 

FAMILrlARIZAT'ION-WI-TH-TERMINAl; 

INTRODUCTION to CONTROL AIDS SYSTEM 


.2- 


IPRACTICE WITH CONTROL AIDS SYSTEM 


3 


PRACTICE WITH CONTROL AIDS SYSTEM 


4 


PRACTICE WITH CONTROL AIDS SYSTEM- 


Ll 


POST-TEST WITH CONTROL AIDS SYSTEM 
(NEW TASK DOMAIN) 



FIGURE 7*. Training Sequence for Control Subjects 
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The initial session begins in the same way as in the experimental condi- 
tion, with a session in which the student is first given some practice using 
the touch panel of the PLAtO terminal by playing tic-tac-toe. This is followed 
by a two part PLATO lesson oh the functions of the control "Aids" system. 
As with the experimental group students, each part of this lesson is fo[I lowed 
by a quiz which the student must pass in order to progress* This introduction 
is followed by a short practice session using a very simple learning task. 

In the second, third, and fourth sessions, the student solves complex troubl 
shooting problems (one for each session) using the control "Aids" system. 
Each of these tasks involves a different problem with the same type of device, 
the Sentence-Generator. These sessions provide practice for the student in 
the use of the control "Aids" system and in troubleshooting problems on 
devices of the sort used for these exercises. In the post-test (Session 5), 
students are to use whatever learning skills they acquired during their 
training to perform a troubleshooting task in the new domain of the essay 
generator. Several types of data are collected during this session, on both 
control and experimental subjects. 

Data Collection 

The data collected during the post-test were designed to measure both 
effective learning and self-directed learning. Effective learning is defined 
in terms of the time required to perform, the task and the number of errors 
made in performing it. For each student data is collected on the number of 
erroneous attempts made to solve the problem and the total time taken to solve 
the problem after being presented with it. Self-di recced learning is much more 
difficult to measure. It was decided that self-directed learning is typified 
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^>by_Jwo_f)henor^^^^^ selectivity in the use of infomatibn sources. 

The dataxollected reflect operational definitions of these phenomena* 
Planning. - 
It is not an easy matter to discover whether a student is engaged in 
effective planning- One type of data saved by our PLATO program is the 
sequenpe in which the student accessed the informatibn resources available 
to him. Our analysis of the troubleshooting task presented to the stuiients ' 
in the post-test session has resulted in the formulation of a set of rules for 
scoring deviations from the order in which the information sources should be 
accessed. These rules, which we call anti-precedence rules, take the fonn of 
prohibitions of certain sequences. The extent to which a student has departed 
from sequences permitted by an ideal task analysis can be.expres.sed;in terms o^^ 
the number of times the student's study sequence violates the;^anti-precedence 
rules. , ^ 

Here is the set of anti -precedence rules :based on our analysis of the task 
used in the post- test: " * 

1* No information source should precede 7^ . 

2- h h should not precede 20 

3- 3^, 4, 6, 8, 15, 21 should not precede Jl 

4- 6,, 11 should not precede 3_ 
6^, 21 should not precede 4^ 
6^, 21 should not precede 8^ 
6_, 21 should not precede 15 

(Note: If the student violates more than one of the rules 
of #4, only one violation is counted.) 
5, 18 should not precede 6^, 21 
• » 
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Here is; an example of how the scoring was done* Consider the foil pwinjg ^ 
hypothetical sequence of accesses, to information sources: 

j 2, 7, 21, 20, 4, 15, 13, 7, 8 
Rule; 1 is violated once, because information source 2^ precedes 7,. Rule 2 is 
also violated once, because information source 2^ precedes sou>"ce 20., Rule .3' 
is violated twice; information sources 2]_ and 15_ both precede J^* Rule 4 is 
violated; 21_ precedes both information sources 15^ and 8. As the note above 
explains, this is counted as only one violation. There are therefore a total 
of five violations of our anti -precedence rules, in the example sequence shown. 

Selectivity in the use of information resources 

Selectivity has to do with the ratio of the use of relevant information 
sources to the use of all information sources. A student for whom this ratio 
is high has read primarily only relevant sourcesT' Three different ratios are 
computed by our program. The first is the ratio of number of relevant infor- 
mation sources read to total information sources read.^ The, second is the 
ratio of the number of readings of relevant information sources to the nimiber 
of readings of all information sources. The third is the ratio of time spent 
reading relevant information sources to the time spent reading aliT information 
sources. 
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Results - 

Mean scores on two measures for the effectiveness of the two qroups 
of learners are presentee! In table !. In the final test session, in which 
students, were required to troubleshoot a faulty essay generator, those 
Students who had- not been exposed to .the training in self-rdirected learning; 
were slightly slower than those who had received the training. The experir ' 
mentaT group subdects, on the average, solved the problem .9. minutes before - 
the control subjects. The number of erroneous choices made "by the two groups 
of subjects before identifying the appropriate component as defective was 
about the same. 

In Table 2 the evidence concerning the selectivity displayed by 
students trained under the two conditions is presented, the measures of 
selectivity that are ratios of the use of relevant information sources to 
total information sources show little or no difference between the two 
groups. Control subjects chose more than twice as many titles to read than 
did the experimental subjects, suggesting that students in the control 
condition were not as selective; however, this difference was not statisti- 
cally 'sipifi cant. 

Table 3 summarizes the measure used to detect planning. Planning, as 
described above, is evidenced by few violations of principles of efficient 
sequencing in reading the available materials.. The means suggest that the 
experimental subjects were better planners than the control subjects, since 
they made only 72% as many planning violations. Again, this was not con- 
firmed statistically. 

A one-way analysis of variance between performance of the two groups 
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Table 1 



Effectiveness of Learning in Post-test Session 

Means 





Time to complete 
(minutes) 


Errors 


Experimental 
(n « 7) 


65 
(31.90) 


2.9 

t3.T3|, 


Control 
(n » 4) 


74 
(25.15). 


3.0* : 

(0.00) 




\ 

* n = 2 




Standard deviations are in parentheses. 
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Table 2 



Selectivity in Post-test Session 



Means 



Experimental 

(n =• 7) 



Control 
(n = 4) 



Titles 
Chosen 



«2 



9 

(3.79) 


0.73 
(0.11) 


0.75 
(0.09) 


^ 

0.73 
(0.12) - 

i 


20 
(13.89) 


0.83 
(0.15) 


0.72 
(0.12) 


• 0.72 
(0.08)^ 



R, = Ratio of number of relevant information sources read to number of \ 
total information sources read 

Rp - Ratio of number of readings of relevant information sources to 
number of readings of all information sources 

R^ - Ratio of time spent reading relevant information sources to time 
spent reading all information sources 



Standard deviations are in parentheses. 
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table 3 




i 


- 


Planning in Post-test Session 










Mas M ^ 








- 


Violations of efficient 






r- 




sequencing 














] 




Experimental 


■ 1.8 




% 




(n = Z) 


(3.08) 




\ 


• 


Control 


2.5 




X 




(n = 4) 


. (5.00) 




\ 




Standard deviations are in parentheses. 
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Indicated that the groups do not differ signiflpantly on the basis of time 
to perforin the task, errors made, selection of relevant titles, and efficient 
sequencing. The difference In the number of titles chosen (Table 2) 
l^proaches signigicance, £.< 0.1. 

I 




Discussion 

» ' ^ — ---^^ 

Jnt§rpretatip'p of these results is problematic. A cursory inspection 
of tfie results leads one to suspect that the special training received by 
'the^experimental group did not have any important effects, and is therefore 
not a|useful\approach to take. Although the students in the experimental 
group seem to-be slightly-more Efficient planners in: the post- test session^ 
and slightly more selective readers, they doh*t seem to be significantly 
more efficient learners. They made about as many errors as did the students 
in the control group and they solved the troubleshooting problem in only 
slightly less time. 

A closer examination of the students' behaviors in the post-test session, 
however, reveals that the nominal experimental treatment may not have been 
operational. The results cannot be interpreted as evidence that the use of 
the self-directed aids system is not helpful, because the experimental sub- 
I jects were not really using the aids system. Only three of the seven experi- 

I ^ ~ mental treatment students ever specified dependencies among information 

sources that they had chosen as relevant. Only two of them ever looked at 
their goal stacks. .None on these students ever specified a dependency be- 
tween objectives. A majority of these subjects (four of the se^iT)^failed 
to formulate more than one objective. (Those who formulated only one 
objective simply restated their task in the form of an objective; e.g,, 
^^•^Identify the defective part pf the Essay Generator.") Thus, the two groups 
did hot really differ in functional treatment. 

V In retrospect, we are impressed that the students in the experimental 
group were able to do as well as those in the control group, the control 
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group students had a much easier assignment. They simply had to make use 
of an automated table of contents to read information sources they .thought 
might help them to solve the problem. What they had to concentrate on was 
learning new information relevant to their task at hand, and all. their 
intellectual resources could be devoted to this task* The; students in the 
experimental group, on the other hand, had a much more difficult assign- 
merit. Not only were they required to solve the same cornplex Teafni^^^^^ 
problems that the control subjects had to solve, they were also; Required to 
;Teapn>and use the many details of the very complex system that was 
to aid them. Under these circumstances, it is surprising that the exper^i- 
mental subjects were able to complete the task in slightly shorter times ' 
than the control subjects, since the experimental s had so much additional 
apparatus to manipulate. Many subjects made it clear to the experimenters, 
both in verbal comments at the conclusion of the post-test and in the written^ 
critiques that were solicited from them that they had not fully 'internalized 
a set of rules for the use of the aids system and: that they were very con- 
fused about the functions of its basic components. In fact, some students, 
after muddling through a number of information sources, simply ^began- to make 
wild guesses about which component might be defective. In some cases, 
students chose as the defective component devices for which they had not 
even read the information sources. 

Not all students who were given the aids: system found it to be useless 
or even a handicap, however. The subject JR,'for example, made very effective 
use of it. She showed good planning by formulating useful objectives and then 
selecting information resources that could help her attain those objectives. 
By the measure of planning discussed in the results, her planning was perfect; 
sfie had no viplations of our rules for efficient sequencing. She was also a 



selective user of infonnation resources* She chosp only nine titles for 
study:, and her selectivity ratios (explained in Table 2) were very hi^^gh 
(R^ p>89, = .90, 0-88) • She was also , an efficient learner. She 
took about an average amount of time to solve the problem; however, unlike 
many other students, she made no errors* She correctly identified the 
defective component on the first attempt. \ 

The fact that this student was better able than others to exploit 
the functions of the automated aids system dramatically highlights the 
variation found in student performance. , An examination of the standard 
deviations given in Tables 1, 2, and 3 confirms this variation. The large 
variation and the small sample size cause any differences between the group 
means to be non-significant. 

Perhaps, then, the fault lies not with the automated aids system it- 
self, but rather with the training program that was designed to teach the 
experimental subjects how to use the aids system. A regression of scores 
on the Nelson-Denny test of reading ability on time taken to complete the 
task reveals an interesting difference between the experimental subjects 
and the control subjects. This difference is shown graphically in Figure 8; 
Note that the control subjects display the relationship that would be expected 
a priori : students who score lower on the reading test take longer to com- 
plete the task. Experimental subjects, on the other hand, show considerably 
less effect of reading ability. However, experimental subjects scoring in. 
. the low range on the Nelson-Denny test require much less time to complete the 
task than control subjects scoring in this same range. Perhaps the automated 
aids system benefits poor readers to a greater extent than it benefits good 

, 

readers. 
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Nelson-Denny Test Scores 



Figure 3. Regression of Reading Scores on Time 
for Each Group 
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VI. DIRECTIONS FOR FUTURE RESEARCH 



A more thorough training program is certainly called for. Students 
in the experimental group were exposed to a very large number of aids-system 
functions, but were given little opportunity to practice using most of these 
functions. Each function was demonstrated once in training, and the student 
was then required to mimic its use once. For many functions, this was the 
only time that the student had to use that function. In our revised train- 
ing system for the use of the automated aids system, students will be required 
^ to practice with each of the available functions until the use of 'each is 

well -understood and easily executed by the subject. 

A second, major problem in our experiment, in addition to the lack of 
adequate practice for those in the experimental group, was the burden of 
learnint; about two complicated systems— the automated aids system and the 
sentence generator— at the same time. It is very important that subjects 
should receive training drill on the use of the^ functions of the aids system 
in a context in which theylare not burdened with the simultaneous^ need to 
puzzle out the workings of another complex system-at the same time. To this 
end, students in future experiments will be drilled on system functions in 
the context of simple learning problems first. Only after the functions 
seem to b^e-well understood will students be required to use the system to 
\ solve thf more complex kinds of problems for which the system was designed 

to bemused. Because the system is really designed to aid in the solution of 
complex problems, its use for simple practice problems may seeiri superfluous 
to students. Our training will therefore contain explanations that the 
simple problems are used for illustration and practice. The student will 
be reminded that the system is most useful for the solution of complex 
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probliems in which large amounts of Information are available, and that the 
simple prpblems are included only for pedagogical reasons. 

A third major problem' in the training our experimental subjects received 
Was that it did not adequately motivate them to make use of the system 
inunctions. Three measures will be taken to increase this -kind of motivation 
in subsequent experiments. First, the reasons for the inclusion of particula 
-functions will be explained more fully to the subjects. They- will be shown . 
hbw each function can contribute to the solution of a problem and under what 
conditions the students will have extra difficulty if he fails ta make use 
of a function. Second, the students will be induced tp improve their trouble 
shooting performances. In realworld on-the-job contexts, professional 
troubleshooters understand that errors in diagnosis are expensive. The 
replacement of properly functioning components is wasteful of both tim and 
materials. Our subjects must be induced not to employ a rr ^o"^ guessing 
strategy to identify the defective component in a, troubleshooting problem. 
Some costs to the subjects, possibly monetary, will be instituted in order 
to prevent the adoption of such a strategy. On the other hand, it is very 
important that subjects not be encouraged to be too conservative in their 
approach. A troubleshooter's time has value, and we do not want to drive 
subjects to a cautious study of all the information resources available to 
them before they make a judgement. Indeed, an important part of our con- 
ception of self-directed learning is that such learning is selective. We 
may, therefore, find it necessary to make use of monetary disincentives for 
reading too many information sources. The third measure we will take to 
increase the students' motivation will be to institute some kind of reward 
system for the use of certain functions of the aids system. In particular. 
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we would Id ke to reward the use of those functions that help the student to 
monitor his progress toward the accomplishment of his task^ such as the 
check-off and X-off functions of the Relevant Contents and Objectives pages* 
Moreover, students would be rewarded for accessing the .goal stack^.in order 
to plan a course of study i Ideally, the administration of rewards for the 
use of such functions sJ|Ould be under the control of the subject himself • 
By following the principles of behavioral self-control set forth in Kanfer 
& Goldstein (1975), Mahoney (1974), Mahone)^ & Thoresen (1974), Thoresen & 
Mahoney (1974), and Watson & Tharp (1972), we should be able to help 
students instill learning habits that they can apply outside of the experi- 
mental environment as well as within it. 
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